fix(gate): never leave the Gittensory Gate check stuck in_progress by JSONbored · Pull Request #655 · JSONbored/loopover

JSONbored · 2026-06-13T22:20:28Z

Production incident: Gittensory Gate stuck `in_progress` for ~8h (PR #650)

Root cause

maybePublishPrPublicSurface posts the "Gittensory Gate is evaluating" check (status: in_progress) early, then does a block of D1 + GitHub + Gittensor work before the completing PATCH — with no request timeouts and no failure finalization. If anything in that window hangs (a bare fetch to a slow/half-open upstream) or throws, the completing PATCH never runs and the check is orphaned in_progress forever; the caller's .catch only logs it.

Confirmed from prod (audit_events / webhook_events): miner detection cache_hit 13:35:08 → pending posted 13:35:11 → then nothing (no completion, no error row → a hang, not a caught throw), while prod D1 was intermittently overloaded. Since miner detection was cached, the hang was in the completion's own un-timed GitHub calls.

Fix (3 commits)

Bound every external call in the gate window so a hang becomes a catchable error: AbortSignal.timeout on the GitHub App fetches + the Octokit instance (src/github/app.ts, 12s) and the Gittensor API client (src/gittensor/api.ts, 10s).
Always finalize the check. Wrap the pending-post → completion window in try/catch; on any failure, finalize the same check run to a neutral, non-blocking "Gittensory Gate — could not finish evaluating — will re-run" state (createOrUpdateErroredGateCheckRun) + a github_app.gate_finalized_on_error audit. Guarded so it never clobbers a real success/failure verdict.
Dead-letter the queue. The gittensory-jobs consumer had no dead_letter_queue, so a webhook job that exhausts its retries was silently dropped. Route exhausted jobs to a new gittensory-jobs-dlq landing queue (already provisioned on the account; no consumer, matching the house pattern) and set max_retries explicitly. Validated with wrangler deploy --dry-run.

Net effect: a transient blip leaves a completed (harmless) check that re-runs on the next push, and a persistently-failing job lands in the DLQ for inspection instead of vanishing.

Test

A queue test drives a confirmed in_progress post, fails the completion PATCH (500), and asserts the same check id is then PATCHed to status: completed, conclusion: neutral, title: "Gittensory Gate — could not finish evaluating", with the audit row recorded. Full suite green; coverage holds above the 97% gate; Workers-runtime tests pass.

Immediate remediation for PR #650

Re-trigger the gate (push any commit, or tick "Re-run Gittensory review" in the panel) — the orphaned run is owned by gittensory[bot], so only a fresh run from the app supersedes it.

Production incident: the 'Gittensory Gate is evaluating' check on PR #650 stayed in_progress for ~8h. Root cause: maybePublishPrPublicSurface posts the pending (in_progress) gate check, then does D1 + GitHub + Gittensor work before the completing PATCH — with no request timeouts and no failure finalization. If anything in that gap hangs (a bare fetch to a slow upstream) or throws, the completing PATCH never runs and the check is orphaned forever; the caller's .catch only logs it. Confirmed via prod: miner detection was a cache hit at 13:35:08, the pending check posted at 13:35:11, then nothing — no completion, no error row (a hang, not a caught throw), while prod D1 was intermittently overloaded. Fix: - Bound every external call in the gate window with a request timeout so a hang becomes a catchable error: AbortSignal.timeout on the GitHub App fetches + the Octokit instance (src/github/app.ts) and on the Gittensor API client (src/gittensor/api.ts, the single fetchJson chokepoint). - Wrap the pending-post -> completion window in try/catch. On any failure, finalize the SAME check run to a neutral, non-blocking 'could not finish evaluating — will re-run' state (createOrUpdateErroredGateCheckRun) and audit it, so the Gate is always terminal and never hangs. Only finalizes when a real conclusion was not already published (no clobbering verdicts). Test proves a failed completion PATCH is followed by a neutral finalize of the same check id. Coverage holds above the 97% gate.

ghost · 2026-06-13T22:20:44Z

Note

Gittensory Gate skipped

PR closed before full evaluation. No late first comment was created.

Signal	Result	Evidence	Action
Gate result	⚠️ Skipped	#655 is no longer open.	No action.

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

ghost · 2026-06-13T22:21:00Z

gittensory · advisory review

Reviewed 5 changed file(s) — two independent AI reviewers.

Suggested action: ✅ Safe to merge — both reviewers found no blocking issues.

Reviewer A · gpt-oss-120b — recommends ✅ merge
The PR adds robust timeout handling for GitHub and Gittensor API calls and introduces a safe fallback that finalizes a pending Gate check to a neutral state on errors, preventing in_progress hangs. The changes are well‑scoped, include comprehensive tests, and respect the public/private output boundaries.

Suggestions

Document the new GITHUB_FETCH_TIMEOUT_MS and GITTENSOR_FETCH_TIMEOUT_MS constants in the developer guide.
Add a fallback for environments where AbortSignal.timeout may not be available, e.g., a manual AbortController with setTimeout.
Consider exporting the timeout constants from a central module for easier configuration.
Add a unit test ensuring that non‑timeout fetches (quick calls) still succeed without being aborted.

Worth double-checking

AbortSignal.timeout is relatively new; verify runtime support on all target Workers environments.
The new timeout may cause legitimate long‑running API calls to be aborted earlier than expected; ensure this does not affect other features.

Reviewer B · nemotron-3-120b-a12b — recommends ✅ merge
The PR adds timeouts to GitHub and Gittensor API calls, introduces a function to finalize stuck Gate checks to a neutral state on error, and wraps gate evaluation in try/catch to prevent in_progress hangs. It also adds a dead letter queue for observable failures and includes a test verifying the fix. The changes are focused, well-tested, and address the core issue without violating public/private boundaries.

Suggestions

Consider adding a comment explaining why 12s and 10s were chosen for timeouts (e.g., based on observed latency percentiles).
The audit event detail uses errorMessage(error); ensure this function doesn't leak sensitive info (though it's internal).
In the try/catch block, gateFinalized is set only when gateCheckResult?.kind === 'published'; verify this aligns with all possible successful gate check outcomes.

Worth double-checking

Timeout values might need tuning if legitimate API calls occasionally exceed these limits (monitor for false-positive timeouts).
Dead letter queue requires operational awareness to inspect failed jobs; ensure team is aware of this new visibility mechanism.
The try/catch now catches all errors during gate evaluation; confirm no unintended errors are silently converted to neutral checks (though audit logs and re-throw help).

superagent-security · 2026-06-13T22:21:01Z

Superagent didn't find any vulnerabilities or security issues in this PR.

… them silently The gittensory-jobs consumer had no dead_letter_queue, so a webhook job that fails its retries was silently dropped (no record once webhook_events isn't reached). Route exhausted jobs to a new gittensory-jobs-dlq landing queue (created on the account; no consumer, matching the house pattern) and set max_retries explicitly. Validated with wrangler deploy --dry-run. Co-requisite infra (already provisioned): `wrangler queues create gittensory-jobs-dlq`.

PR #655 finalizes the pending in_progress Gate check on any failure that throws. But a 403 on the completion call is classified as permission_missing and returns without throwing, so the catch never runs and the pending check is orphaned in_progress forever. The pending check already posted (pendingGateCheckRunId is set), proving the App had Checks:write, so a 403 at completion is almost always a transient secondary-rate-limit rather than a real revocation. Finalize the pending check to neutral in the permission_missing branch too (mirroring the catch); a genuine revocation makes this PATCH 403 as well and it is swallowed. Closes #730 Co-authored-by: ghost <49853598+JSONbored@users.noreply.github.com>

github-project-automation Bot added this to gittensory - v1 roadmap Jun 13, 2026

github-project-automation Bot moved this to Todo in gittensory - v1 roadmap Jun 13, 2026

dosubot Bot added the size:L label Jun 13, 2026

ghost added the gittensory:reviewed label Jun 13, 2026

ghost added the gittensory-review label Jun 13, 2026

JSONbored self-assigned this Jun 13, 2026

Merge branch 'main' into fix/gate-finalize-on-error

7735cf0

JSONbored mentioned this pull request Jun 13, 2026

fix(db): timestamp columns store a real ISO value on omit, not the literal string #656

Merged

JSONbored merged commit 541559e into main Jun 13, 2026
11 checks passed

JSONbored deleted the fix/gate-finalize-on-error branch June 13, 2026 23:09

github-project-automation Bot moved this from Todo to Done in gittensory - v1 roadmap Jun 13, 2026

JSONbored mentioned this pull request Jun 13, 2026

feat(db): scheduled data-retention pruning for log/snapshot tables #660

Merged

galuis116 mentioned this pull request Jun 14, 2026

[Bug]: transient 403 on Gate completion leaves check stuck in_progress #730

Closed

philluiz2323 mentioned this pull request Jun 14, 2026

fix(gate): finalize the Gate on a transient 403 at completion #732

Merged

Navigation Menu

Uh oh!

fix(gate): never leave the Gittensory Gate check stuck in_progress - #655

fix(gate): never leave the Gittensory Gate check stuck in_progress#655
JSONbored merged 3 commits into
mainfrom
fix/gate-finalize-on-error

JSONbored commented Jun 13, 2026 •

edited

Loading

Uh oh!

ghost commented Jun 13, 2026 •

edited by ghost

Loading

Gittensory Gate skipped

Uh oh!

ghost commented Jun 13, 2026 •

edited by ghost

Loading

Uh oh!

superagent-security Bot commented Jun 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

JSONbored commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Production incident: Gittensory Gate stuck in_progress for ~8h (PR #650)

Root cause

Fix (3 commits)

Test

Immediate remediation for PR #650

Related

Uh oh!

ghost commented Jun 13, 2026 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Gittensory Gate skipped

Uh oh!

ghost commented Jun 13, 2026 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

superagent-security Bot commented Jun 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

JSONbored commented Jun 13, 2026 •

edited

Loading

Production incident: Gittensory Gate stuck `in_progress` for ~8h (PR #650)

ghost commented Jun 13, 2026 •

edited by ghost

Loading

ghost commented Jun 13, 2026 •

edited by ghost

Loading